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Using This Guide 





Section 1: Tools for District and School Leaders is aimed at 
district and school leaders who are just beginning to think hard 
about acquiring an electronic data system. Designed to support 
each person’s thinking, this section contains discussion elements 
along with corresponding questions that prompt dialogue around 
key issues for readiness. 

Section 2: Tools for Facilitators makes it possible to use the 

guide as a resource for a group process to ascertain readiness — a 
sort of needs assessment. This section provides additional 
information on choosing and implementing a data system that will 
help guide further investigations, as well as worksheets to help 
facilitate or organize a discussion. 

Section 3: Resources presents a variety of resources for data use 
and data systems. 



Tell Us What You Think 



The ESA Alliance is interested in hearing how this guide is 
working for you. If you have any feedback or have used this guide 
in a creative way that you would like to share, please e-mail your 
comments to datause@leamingpt.org . 
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Setting the Context for Change 



Data fuel the process of change. Schools and districts should have electronic data systems that 
store data and make it easy to extract useful information. You and your staff have questions; you 
need answers. Your data, when stored in a suitable electronic data system, will help you 
construct better answers to your questions more quickly. Do you know what your questions are, 
what they will be? That is, do you really know why you want a data system? 

Electronic data systems are complex and expensive. They 
consist of numerous components, pieces, and functions: 
data warehouses, data marts, data dictionaries, extraction- 
translation-loading (ETL) tools, data cleansing, analytics, 
user portals, decision support, identity management, and 
more. Rather than starting with these specialist’s terms, this 
document begins with your questions: 

• What do you want the system to do for you? 

• Are you and your organization ready to support a 
system that does what you want? 

• Will you be able to act on what it tells you? 

• What is required to get you and your district to that point? 

This guide will help you think about what you want from an electronic data system. It will help 
you identify the right questions to ask about a data system and about your district’s readiness and 
capacity to use it effectively. As you evaluate your responses, you will better understand what 
you already know and what you have yet to leam, as well as the context within which you will 
put that knowledge to work. After all, you will not receive good value on investments in these 
systems if you and your staff are not clear about what you expect, what you want, and what your 
questions are. 



As the late statistician John W. Tukey, 
Ph.D., (1962) put it, “Far better an 
approximate answer to the right 
question ... than an exact answer to 
the wrong question” (pp. 13-14). 

Decision makers want answers. Answers 
and decisions both begin with questions. 
A data system that saves and stores 
your data — and has easy-to-use tools to 
render that information promptly and 
meaningfully — greatly increases the 
likelihood of better answers more often. 



Section 1: Tools for District and School Leaders is 

organized around eight discussion elements — Will, Scope, 
Risk, Readiness, Information Quality, System Quality, 
Impact, and Cost. Section 2: Tools for Facilitators is a set 
of facilitation materials consisting of personal reflection 
worksheets, a discussion worksheet, and a planning 
worksheet. Section 3: Resources is a brief annotated 
bibliography of resources for data use and data systems. 

Data systems are “back-office” tools in most organizations 
and businesses. Space up front is given over to showing what 
the business or organization is about. The same is true in 
schools and districts: The data system should support what 
you are trying to accomplish; it should not be what you accomplish (unless, of course, your title 
is chief information officer or data systems manager). 



It may be wise to accept Dr. Tukey’s 
(1986) counsel one more time: 

“The combination of some data and 
an aching desire for an answer does 
not ensure that a reasonable answer 
can be extracted” (pp. 74-75). 

A data system and good questions will 
help; however, they are no guarantee. 
You, not the data, will make the decision. 
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Data make change visible. Science begins with doubt; its hallmark is suspicion (Powers, 1999). 
Scientists look at evidence, draw inferences, modify what they do, look at the evidence again 
(Platt, 1964). They continue these repetitions, these trials and errors, these experiments, looking 
for improvements and are guided by them. Careful data records support accurate inference from 
observation and second warrants for interpretation. In education, we speak of a culture of 
inquiry, of continuous improvement cycles, not of experiments. However, the cycle of 
observation, interpretation, and controlled trials is very much the same. A data system 
contributes by helping us to store, track, and display relevant data. That’s all it does, but that’s 
quite a lot. 

The following pages in this section contain information on the following key elements: 

• Will 

• Scope 

• Risk 

• Readiness 

• Information Quality 

• System Quality 

• Impact 

• Cost 

For each element, there is an introduction, questions, and suggestions for interpretation of 
responses. 
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Element A: Will 



Introduction 

A review of data should lead to action. If you are not prepared to do something different, then all 
the data in the world will not matter. 

According to Mel P. Heller, Ed.D., Professor Emeritus of Educational Leadership and Policy 
Studies at Loyola University of Chicago, “It is permissible to move ahead without the answer; 
however, if you do not have a question, do not begin.” Heller also reminds us that “waiting for 
the perfect solution is paralysis.” 

Questions are more important than answers. You do not need all the answers to start; in fact, 
waiting for all the answers means you will not start. Decide where you want to go. Begin going 
there. Ensuring that you have the necessary will to get there is critical. 

Implement small, but have a larger plan. If you begin to invest and build a data system driven by 
where you are without thinking of where you want to be, you will likely find in the not-so-distant 
future that you need to start over, again. 

The purpose of an electronic data system is to help you navigate changes to monitor 
improvement efforts. But that contribution is only a part of a larger drive toward continuous 
improvement. Adopt and work a continuous-improvement cycle. It matters less which cycle than 
that all staff subscribe to it. Acquiring a data warehouse or a data analysis tool, including 
investing in staff training in data-driven decision making, is really about efficiency, feedback, 
and taking action. By placing the data in front of more staff, you also move responsibility for 
action — and the rationale for taking action — down to the implementer, the doer. You, as leader, 
can therefore spend your time on determining how to support the good decisions and good work. 
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Questions Related to Element A: Will 

A1 . What will you do if your data tell you that some students are learning but others are not? 
Q Nothing 
Q Something 
Q Do not know 

A2. Will you be willing to replace or modify instructional programs or strategies currently 
in place? 

O No 
O Yes 

O Do not kn ow 

A3. Will you let staff members make their own decisions about instruction? 

O No 
Q Yes 

Q Do not know 

A4. Will this data-system effort become a permanent way of doing business? 

O No 
Q Yes 

Q Do not know 

A5. Will the data system be part of a quantitative continuous-improvement process to help 
you monitor progress to support better decisions? 

Q No 
O Yes 

O Do not kn ow 

Interpretation of Responses 

A “Nothing” answer to the first question and more than two “No” answers suggest you do not 
yet have the will to act upon what the data may tell you. An investment in an electronic data 
system will probably not help you improve the instructional effectiveness of your school or 
district. 

A “Something” answer to the first question and at least two “Yes” answers confirm that you have 
the will. As always, the results will be in the details. Please turn the page to continue your 
efforts. 

Any other pattern of responses suggests you should think more about why you need a data 
system. You’re probably going to benefit from having one, but in order for it to make a strong 
difference, you first may need to increase your own commitment. 
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Element B: Scope 



Introduction 

Order matters: the chicken, the egg; the cart, the horse. Whether the chicken or the egg came first 
is moot. Chickens didn’t evolve from eggs or eggs from chickens — the two came together; the 
question has little practical application. In the real world, however, placing the horse before the 
cart works best. In real life, an answer without a question is not particularly useful. A goal 
without some questioning is dangerous. We can be in a hurry to collect and present data in hope 
of finding an answer, but if we never state a question, any answer will do. 

Begin with a goal, with questions such as “What will learners be able to do when they leave this 
school?” The term learner helps focus on what schools are about. If you ask the average person 
how to define or recognize a school, the typical response is “where there are students and 
teachers.” However, it isn’t students and teachers that define a school; it’s learning. 

What should students be learning? How? Why? Today, an easy answer is that the learner will be 
proficient on the state test. This is where the pain is, especially for schools not meeting adequate 
yearly progress (AYP). But it is hardly clear how being proficient on a test equips the student for 
life. 

The No Child Left Behind (NCLB) Act is driving districts towards data-driven decision making. 
If “How are we doing in terms of NCLB?” is the only question, then your data needs are very 
specific. A bigger question for most districts is “How will we know that a student can [insert the 
district’s mission, values, curriculum, etc.]?” If these are the kinds of questions on which you 
will focus, you will need to think carefully before you embark on building a system of data 
collection and analysis. These are big questions. You will almost certainly answer them in 
smaller pieces. Your data system will have to be able to capture numerous data elements as well 
as help you answer “big-picture” questions. It will need to be flexible enough to grow with you 
and yet not be cumbersome to use. 

A good starting place is to define what the end product should do. Given your questions, describe 
or illustrate the kinds of answers you expect to help you decide what to do. Do not think of the 
minimum necessary. Describe useful answers and work out incremental steps to get to the goal. 
Data-driven decision making is about making sure you keep taking the right steps. 
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Questions Relating to Element B: Scope 

B 1 . Do you know the kinds of questions you will be trying to answer? 

O No 
O Yes 

O Do not kn ow 

B2. Will these questions be reasonably precise and become more precise as time goes by? 

O No 
O Yes 

O Do not kn ow 

B3. Will there be other kinds of questions? 

Q No 
Q Yes 

Q Do not know 

B4. Will students, parents, and community also be able to use (parts of) the system? 

O No 
O Yes 

O Do not know 

Interpretation of Responses 

A “No” answer to the first question or “Do not know” answers to the other questions should stop 
you: More thought about what you want to know is necessary. 

“Yes” answers suggest you are ready to move forward. 

If you also can answer “Yes” without the words kind of, a more limited, more tightly configured 
system may be all you need now because your needs appear tightly focused. Still, be careful. 
Investments in data systems of greater generality are usually the better choice. On the other hand, 
it is possible to follow a pathway of growth over time in system functionalities. It is not 
necessary to do (or buy) everything at once. Discuss these matters carefully with system vendors 
to prevent dead-ends down the road. 

The last question raises other issues. You probably will need to engage students’ entire network 
of support, including parents, community, and the students themselves. That implies wide but 
controlled access to the data system. It probably also implies that the data system will not be a 
single purchase, but rather grow into a “stack” of applications — that is, software components that 
provide a mix of well-integrated functions. 
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Element C: Risk 



Introduction 

In mutual fund and stock reports, the following footnote is common: “Past performance is no 
guarantee of future results.” Nevertheless, we make predictions all the time. Yesterday’s data 
lead to conclusions about tomorrow. History is interesting, but a teacher’s concern is the students 
in the room today and where they will be in June. The key question becomes “How confident are 
we of the predictions?” Data should paint a trustworthy path to the future. 

Prediction should be modeled — that is, shaped by instructional choices carefully captured in data 
and rigorously analyzed. This is possible in schools and should be done more often. Careful 
modeling is all the more important when we recall the uncertainty in even a key outcome 
measure, standardized tests. In the line print of the student reports sent out by test publishers, it 
says that individual scores fall in a range, not at a point. That range is surprisingly large, often 
more than 10 percent of the total scale or more than a half-year’s growth. Given such built-in 
uncertainty, modeling is necessary to “tune in” strong and trustworthy signals from the data. 
Decisions should be based on the signals in the data, not on the noise. 

State and other large-scale tests lack the sensitivity to paint a detailed picture of a student’s full 
knowledge and skill. Because of volume, they are better at painting a picture of a district or 
school. If you need to track individual student progress, use multiple points of data over time on 
multiple measures. Require that the data system inform you of the level of confidence you should 
place in the computed results. One or two data points, no matter how compelling, are never 
enough. A trend line requires at least three data points, not two. 

Teachers tend to thi nk one child, one day, one lesson at a time; district leaders thi nk about 
programs, school years, and beyond. The decisions made and the risks taken are clearly different. 
It is commonly assumed that a district’s chosen data system can speak meaningfully to both. 

That may be true, but not if the data system uses the same calculations, procedures, 
visualizations, and reports for both sets of purposes. Accurate rendering of the relevant meanings 
of the data for each set of purposes requires distinct procedures and specific reporting and 
visualizations. Consequently, risk assessments will vary. 
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Questions Relating to Element C: Risk 

C 1 . Do you want to know how the schools are doing or how the district is doing? 

O No 
Q Yes 

Q Do not know 

C2. Do you want to know how individual students are doing? 

O No 
Q Yes 

Q Do not know 

C3. Do you want to kn ow how the schools or the district will do next year and subsequently? 
O No 
O Yes 

O Do not kn ow 

C4. Do you want to know how individual students will do next year and subsequently? 

Q No 
Q Yes 

Q Do not know 

C5. Can you estimate the amount of risk you run in the decisions you make, 
given the data you use? 

O No 
O Yes 

O Do not kn ow 

Interpretation of Responses 

Chances are you answered (or wanted to answer) “Yes” to the first four questions and “No” to 
the last one. However, keep in mind that while similar data-reporting procedures can address 
both student- and group-level questions, the warrants for decisions about action will be markedly 
different. Aggregation over individuals reduces statistical risk; decisions about individuals 
require multiple measures and multiple data points as well as sensitive interpretation. 

Analytic and visualization techniques exist to provide better, more nuanced answers than those 
supported by simple bar graphs and pairwise comparisons. Prioritize your wants and needs. 
Demand that your data system comprehends and makes use of the complexity of education data 
in ways that increase signal and decrease noise. Require that data systems always indicate 
confidence levels, given data quality and analytic procedure. You always need to know how 
much trust to place in results. 
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Element D: Readiness 



Introduction 

Schools vary widely in the ability of their staffs to use data. Some have built strong cultures of 
data use; others not. What is clear is that data use requires time, training, and continuous support. 
Teacher-training institutions typically do not address data interpretation as a classroom skill. 
Graduate programs for administrators also provide little training in data applications or to 
support staff in learning data skills. 

The phrase “Lies, damn lies, and statistics” was spoken in jest. (The quote often is attributed to 
Mark Twain, who claimed he got it from Benjamin Disraeli, although there is no record of 
Disraeli saying it. Statistics cannot resolve this.) Still, for many, statistics are a history of (t)error 
(Huff, 1954). The problem lies in how statistics are presented. The useful way is to hypothesize 
and test, as simply as this: Identity a problem, pose it as a question, research the question for 
what contributes to it, then measure performance. The size of the difference between 
measurement and benchmark combined with knowledge and experience will help you determine 
what to do. Statistics should help you see whether differences are meaningful. If they do not 
provide that help, turn to another basis for decision. 

Data make change visible. Data provide an empirical lens that magnifies objective detail while 
distancing us from personality. Data can confirm if there is change or not. The smaller, the 
tighter, the more frequent the feedback loops that the data system supports, the more staff can 
make decisions, the more frequently decisions can be made, and the more likely that the 
decisions made will be better ones. 

Data become information through conversations among staff members. These conversations 
reinforce the feedback loops fed by the information flowing from electronic data systems. 
Allowing these practices to take hold and become habitual requires time, training, practice, 
supervision, and leadership. Conversations about data always connect back to individual 
classroom practices or administrative decisions. An environment of mutual professional trust, a 
“safe space” so to speak, is therefore a necessary condition to enable these conversations. 
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Questions Relating to Element D: Readiness 

D 1 . What will be the level of budgetary support? 

Q Fixed, one-time amount 
O Fixed, multiyear amount 
O Flexible, multiyear funding 

D2. Who will be the effort’s champions? 

O None needed 
O Roles defined 
O Selected 
O Everyone 

D3. What technical staff will be needed for system deployment and ongoing support? 

Q Unsure 

Q Role requirements defined 
Q Selected 

D4. Will there be support for staff to ensure that data are used properly and in sound ways? 

O No 
O Maybe 
O Yes 

D5. Who is available to guide the development and delivery of suitable training on the use of 
data for measurement, monitoring, and interpretation? 

Q None needed 
Q Role requirements defined 
Q Selected 

D6. Will there be support to help staff select, design, and implement new instructional 
strategies? 

O No 
Q Maybe 
O Yes 

Interpretation of Responses 

Without technical staff and without user champions (note the plural; note the advantages of 
“everyone”), the data system project will fail to meet expectations. Without training in data use 
and training in instructional applications, the data system itself will not meet expectations. 
Although sufficient time for training and for practice is always necessary, it is often overlooked. 
Without an adequate funding stream, the system’s implementation and its supports will be 
insufficient. The specific details of the implementation work need not be well detailed at the 
beginning, but the will and the resources must be present. 
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Element E: Information Quality 



Introduction 

You have a good sense of your questions about students’ learning. Now you must specify the 
data you will use. Resist making a list of the data you already have. Instead, figure out what data 
you need in order to persuade others to act. 

For example, say that data show performance is well below target, but no one is willing to 
change because, after all, it is only one piece of data. How much more data will you need to 
make your case? What comparisons will compel someone to say that the original idea of low 
performance was correct? 

In the end, if we do not like what the data say, one question remains: Is it bad data or weakness 
in our teaching and learning practices? The important realization is that in either case, we are in 
control, so by default we have to fix it (the data, or the teaching, or the learning) — this is the 
heart of accountability. 

There are many possible challenges to data integrity, validity, reliability, precision, and meaning. 
Getting the technical parts of data quality right is complex, technical, and messy. Seek advice! 

(A good place to begin is the Forum Guide to Education Indicators from the National Forum on 
Education Statistics [2005].) Work to keep the technical details in the service of the larger 
purposes. Focus on the purposes for which you will use data, on the logical and rhetorical 
structures you need to build to make your case. Let technical experts lead the cleaning and 
loading. 

Lay out a logical design for how you will use the data. Researchers call this a theory of action. It 
tells how things connect and how each influences the other. This theory should also describe why 
your school does what it does for teaching and learning. The data then model your beliefs and 
practices. With that modeling, data will help you see clearly what is working, what is not 
working, and where new effort is needed. Let the technical specialists build out the data 
warehouse to house the data elements that are needed to test your theory of action. 
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Strategic Questions Related to Element E: Information Quality 

El . Do you know the information you will need? 

O No 
O Yes 

O Do not kn ow 

E2. Do you know when you will need that in formation? 

O No 
O Yes 

O Do not kn ow 

E3. Will the system readily accept additional data elements? 

Q No 
Q Yes 

Q Do not know 

E4. Do you know what processes will ensure that the data are accurate and valid? 
O No 
O Yes 

O Do not know 

E5. Do you know how to ensure that the data will remain secure and confidential? 
O No 
O Yes 

O Do not know 

E6. Do you know that inferences drawn from the data will be sound? 

Q No 
O Yes 

O Do not know 



Tactical Questions Related to Element E: Information Quality 

E7. Do you know the data elements you have and where they are? Do you know what 
elements you still need? 

O No 
O Yes 

O Do not kn ow 

E8. Currently, do you have designated data owners, caretakers, users? 

O No 
O Yes 

O Do not know 
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E9. Do you know how much data cleaning will be needed? If there is much data, do you 
know the extract, transfonn, and load (ETL) priorities? 

Q No 
Q Yes 

O Do not know 

E10. Do you know how frequently data will need to be updated? 

O No 
Q Yes 

O Do not know 

Ell. Does the vendor’s data update schedule meet your analytic and reporting needs? 

Q No 
O Yes 

O Do not kn ow 

Interpretation of Responses 

Quality ensures that users are confident and will trust the data. Data must be accurate, relevant, 
useful, and sufficient. This must be true at the level at which the data are collected and stored and 
for the re-combinations and re-groupings that will happen subsequently. Ensuring this requires 
sophisticated technical skills — even more so once data fields begin to proliferate, as they will. 

That complexity must be transparent to the persons who will make decisions based on the 
patterns “read” in the data. Achieving transparency requires careful working together of those 
who design the questioning domains (you); those who collect, store, and manipulate the data (the 
technical experts, usually vendors); and your teachers and principals who will convert your “big” 
questions into data explorations that help them see what they need to do. 
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Element F : System Quality 



Introduction 

When introduced, a data system often generates a distinct reaction: “Cool!” What you really 
want, however, is for staff to ask a question and be able to point to data that support an answer or 
a way to an answer. When staff members become inquisitive, you want them to explore — these 
are moments similar to the teachable moments they look for in their own classrooms. 

Staff members become excited using an analysis tool if it delivers what they need or if it answers 
their next question. During vendor interviews, you should ask questions that help you see 
whether the system will induce that excitement. Technical architectural details are all good and 
well, but use will be driven by the conversations staff can hold about the data and their analyses, 
not the technical details. 

If staff members are excited and say things that make you realize they are thinking new thoughts, 
then you have made a good choice. Again, it is about data-driven decisions and actions that lead 
to greater productivity and performance. 

If you want teachers to be active users of the data system, it is critical that the data system’s 
mode of presentation fits transparently into the routines of their daily lives. If the system requires 
30 to 45 minutes of a staff member’s time to construct an analysis or explore a concern, staff will 
not use it much — unless you set aside such blocks of time. Systems that require much shorter 
contact times or present early warning indicators in a dashboard mode probably fit school life 
better, assuming the dashboard lights illuminate points of interest. 

Such capacities ask for considerably more interactivity and programmability of an electronic data 
system than delivery of “canned” reports and graphics, even analytics. The questions that follow, 
therefore, focus on the capacity of systems to be nimble, quick, and useful under routine and 
novel situations. 
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Questions Relating to Element F: System Quality 

F 1 . Does the system interconnect and integrate the various data sources you use? 

O No 
O Yes 

O Do not kn ow 

F2. Does it meet the Schools Interoperability Framework (SIF) standard (a standard for data 
sharing supported by vendor serving the K-12 market) and the Open Database 
Connectivity (ODBC) standard (a data industry standard to let applications access data 
from numerous repositories)? 

O No 
O Yes 

O Do not kn ow 

F3. Does the system store data at the lowest possible grain size, given your needs? 

O No 
Q Yes 

O Do not kn ow 

F4. Are all necessary metadata recorded and usable for queries? 

Q No 
O Yes 

Q Do not know 

F5. Does the system permit flexible filtering and aggregation over units, over time, and 
among data elements? 

O No 
O Yes 

O Do not kn ow 

F6. Does the system provide capable and useful decision support and analytic tools? 

Q No 
Q Yes 

Q Do not know 

F7. Is there a route to enhance or extend the system’s decision support and analytic tools? 

O No 
O Yes 

O Do not kn ow 

F8. Is the system interface friendly, supportive, and informative? 

O No 
O Yes 

O Do not know 
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F9. Does the system interface support progressive disclosure in response to staff inquiries? 

O No 
O Yes 

O Do not kn ow 

F10. Will the system scale readily as user numbers increase? 

O No 
Q Yes 

Q Do not know 

FI 1 . Will system implementation time meet your needs? 

O No 
O Yes 

Q Do not know 

F12. Does the vendor provide adequate technical support? 

O No 
O Yes 

O Do not kn ow 

F13. Does the vendor provide training for instructional staff that connects clearly to your 
school and district mission and procedures? 

O No 
Q Yes 

Q Do not know 

Interpretation of Responses 

You should seek as many affirmative responses to these questions as possible. No product 
currently can meet all these requirements, despite the fact that vendors will promise you their 
data systems will do (nearly) everything you ask. Your own understanding of some of the terms 
in these questions may be somewhat vague or uncertain today. Your skills and knowledge will 
grow. For now, focus on ensuring that product vendors can reply “Yes” to most of these 
questions. For each “Yes,” have the vendors tell you why it matters. In doing so, you will learn a 
great deal about the appropriateness of both the product and the vendor to meet your needs. 

All the currently available electronic data systems for K-12 are adequate for storing and tracking 
student demographics, attendance, grades, assessment, and other basic data elements. Connecting 
these to student scheduling information and teacher attributes is much more difficult; usefully 
interconnecting these data elements to each other over time can also be problematic in many 
products. Thorough study of a product’s reporting and analytic capacities will tell you more of its 
utility for you than close inspection of technical underpinnings. Note whether the analytics focus 
on describing the past or predicting the future. Note carefully whether that description or 
prediction is grounded in raw data or modeled data. You count the latter — and you want to be 
able to see and alter the model parameters. 
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The acid test will be how much the system gets used and the quality of the decisions your staff 
make after use. Involve staff members beforehand in opportunities to review reports and queries, 
to see analyses and talk through their reactions and their predictions of their behavior. Ask them 
to spend time with demo or mockups the vendor may provide. Their reactions about potential 
usefulness and value should be a strong element in your decision making. Use these reactions to 
query the vendors further. Signs of flexibility, support, and meeting your needs straightforwardly 
will be clear markers of good future relations. 



20 — Effective Use of Electronic Data Systems 



Learning Point Associates 



Element G: Impact 



Just what is a data-driven decision? It will vary with the decisions you permit staff to make. 

Make sure all staff members understand what their core responsibilities are and what their 
obligations are for learning to do that work better. Understanding this will make a big difference 
in how staff will seek, manipulate, present, and use data. (A useful support is the Forum Guide to 
Building a Culture of Quality Data from the National Forum on Education Statistics [2004].) 

The key is to understand which data are relevant, how data are structured and presented, and how 
big a difference must be before it provides researchable evidence for action. Are you prepared to 
support the data-driven actions proposed by staff? Will you allow them to experiment 
instructionally with data as their guide? Will you give them the necessary resources to explore 
and understand options and decide well? 

A five-point theory of action supports the drive toward data-based decision making in 
classrooms, schools, and districts: 

• The purpose of data is to make change visible. Data offer an objective lens, magnifying 
details sometimes unseen and distancing the more personal elements. This enhances 
richness of information and reduces ad hominem irrelevancies. 

• Change requires understanding. Schools are “loosely coupled” settings — once doors 
close, staffs are largely in control of their own work. Therefore, each staff member needs 
to come to understand in personal terms what changes are necessary. At the same time, as 
each person arrives at an understanding about change, these understandings need to align 
around the core issues of instructional productivity. When this happens, cacophony 
becomes chorus. 

• Understanding requires feedback. Data can confirm that change is happening, what the 
direction of change is, and what the rate of change is. The smaller (fewer people, fewer 
issues), the tighter (the quicker) the feedback loop, the more frequently decisions can be 
made and the more likely decisions will be accurate. Monitoring becomes continual 
improvement. 

• Information is social. Data must be visible to all. Data must be and remain at the center 
of ongoing conversations about the work and its importance. Data convert into 
information through social interaction. Information is a shared good: It has no value when 
restricted to one person. 

• Change is local. At bottom, change is an individual event: One person acts, then another. 
In schools, it is the school community — the conversing unit of interconnected individuals 
whose work is aligned because of common understanding — that breathes life into change. 
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Element H: Cost 



The costs for acquiring, installing, maintaining, and operating an electronic data system are 
variable and difficult to pin down. In part, this is driven by the fact that data systems are not sold 
as commodities but are individually contracted between vendor and district. While installation, 
maintenance, and operational costs (and profit margins) are quite clear to the vendors, there are 
assorted uncertainties from the vendor’s perspective that lead to variations in costs: 

• Data extraction, cleaning, and loading. Data sources and data requirements vary from 
client to client. Commonalities do exist, such as commercial or state assessment records, 
but even these vary by depth and breadth and use preferences. Data that are digitally 
stored in warehouses today are typically fairly easily extracted, and the situation is 
improving. Nevertheless, local data and local data storage remain a confusing world and 
require considerable up-front effort to identify, structure, convert, and load. 

• Application stacks. A decade ago, a district bought a single vendor’s system and that was 
that. Today, multiple applications live side by side as well as in vertical stacks. Library 
and bus-routing systems are expected to feed select data elements to the instructional 
management data warehouse. Security applications and identity management software 
control user rights and access across multiple applications and multiple interfaces. 
Ensuring transparent connections and immediate access is no small task. Working out the 
relationships with a district’s changing constellation of software and applications requires 
considerable and continuous effort, despite the industry’s successful concentration on 
standards for data, interfacing, and control. 

• Client requirements. Despite the apparent uniformity of school and district needs with 
respect to data and data solutions, vendors are continually surprised by the differences 
among districts in terms of what they want and need from data systems. Vendors do not 
want unsatisfied customers; however, if they do not understand what the customer really 
wants, contracts soon become uncomfortable. 

• Learning area. Software can be installed. To make it useful and reliable often requires 
considerable learning by staff members. With experience, utility should increase. But 
escalating return on investment will depend on your own investment in and patience with 
your staffs learning — not to entertain your own ability to show the way. 

In Data Systems Resources that appear in Section 3: Resources, there are several reports that 
include some pricing examples. See the following resources: College of Education, University of 
Texas at Austin (n.d.); Schools Interoperability Framework Association (2003); and Stein 
(2003). 
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Section 2: Tools for 
Facilitators 




Using the Tools for Facilitators 



Tools in This Section 

This section consists of the following tools: 

• Personal Reflection worksheets for six of the key elements: 

■ Element A: Will 

■ Element B: Scope 

■ Element C: Risk 

■ Element D: Readiness 

■ Element E: Information Quality 

■ Element F: System Quality 

• Discussion/Facilitation Worksheet 1 for collaborative discussion 

• Discussion/Facilitation Worksheet 2 for planning next steps 

Instructions for Using This Section 

Note: Everyone already should have worked through the questions for each element in Section 1: 
Tools for District and School Leaders. 

First, as a group or as individuals interested in a data system, everyone should reflect on the key 
elements using the Personal Reflection worksheets that follow. 

Next, using Worksheet 1, one member should facilitate a conversation around each element, and 
the exploratory team could come up with consensus on what is known and what needs more 
investigation. Alternatively, an individual could use the worksheet to organize his or her own 
thoughts in one place. 

Then, use Worksheet 2 to plan actions for further investigation and dialogue. 
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Personal Reflection on Element A: Will 



Directions: Review your answers and the interpretation for Element A: Will (beginning on 
page 5). Then work through the following questions, and be prepared to share and collaborate 
with other members of the exploratory team. 

Summarize your sense of overall readiness in terms of will for moving forward with making a 
decision about electronic data systems. 



Which questions suggest readiness? Please provide the question numbers and answers. 



Which questions confinn areas of unpreparedness? Provide the question numbers and ideas 
about where someone might look for this information. 
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Personal Reflection on Element B: Scope 



Directions: Review your answers and the interpretation for Element B: Scope (beginning on 
page 7). Then work through the following questions, and be prepared to share and collaborate 
with other members of the exploratory team. 

Summarize your sense of overall readiness in terms of scope for moving forward with making a 
decision about electronic data systems. 



Which questions suggest readiness? Please provide the question numbers and answers. 



Which questions confinn areas of unpreparedness? Provide the question numbers and ideas 
about where someone might look for this information. 
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Personal Reflection on Element C: Risk 



Directions: Review your answers and the interpretation for Element C: Risk (beginning on 
page 9). Then work through the following questions, and be prepared to share and collaborate 
with other members of the exploratory team. 

Summarize your sense of overall readiness in terms of risk for moving forward with making a 
decision about electronic data systems. 



Which questions suggest readiness? Please provide the question numbers and answers. 



Which questions confinn areas of unpreparedness? Provide the question numbers and ideas 
about where someone might look for this information. 
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Personal Reflection on Element D: Readiness 



Directions: Review your answers and the interpretation for Element D: Readiness (beginning 
on page 11). Then work through the following questions, and be prepared to share and 
collaborate with other members of the exploratory team. 

Summarize your sense of general readiness for moving forward with making a decision about 
electronic data systems. 



Which questions suggest readiness? Please provide the question numbers and answers. 



Which questions confinn areas of unpreparedness? Provide the question numbers and ideas 
about where someone might look for this information. 
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Personal Reflection on Element E: Information Quality 



Directions: Review your answers and the interpretation for Element E: Information Quality 
(beginning on pagel3). Then work through the following questions, and be prepared to share 
and collaborate with other members of the exploratory team. 

Summarize your sense of overall readiness in terms of information quality for moving forward 
with making a decision about electronic data systems. 



Which questions suggest readiness? Please provide the question numbers and answers. 



Which questions confinn areas of unpreparedness? Provide the question numbers and ideas 
about where someone might look for this information. 
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Personal Reflection on Element F : System Quality 



Directions: Review your answers and the interpretation for Element F: System Quality 
( beginning on page 16). Then work through the following questions, and be prepared to share 
and collaborate with other members of the exploratoiy team. 

Summarize your sense of overall readiness in terms of system quality for moving forward with 
making a decision about electronic data systems. 



Which questions suggest readiness? Please provide the question numbers and answers. 



Which questions confinn areas of unpreparedness? Provide the question numbers and ideas 
about where someone might look for this information. 
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Discussion/Facilitation Worksheet 1 — Collaborative Discussion 



Directions: After each member of the exploratory team works though each element and personally 
reflects on the questions and interpretations, it will be necessary to work with the team on a collaborative 
response in terms of readiness. For each element, reflect on the following questions to help guide a 
collaborative discussion and further data gathering around the use of a data system. 



Element A: Will 



Summarize the team’s overall response to the survey questions. 



Which of the questions is the district prepared to answer? 



Which questions need more investigation and clearer answers? Where might we look for this 
information? 



Element B: Scope 



Summarize the team’s overall response to the survey questions. 



Which of the questions is the district prepared to answer? 



Which questions need more investigation and clearer answers? Where might we look for this 
information? 



Element C: Risk 



Summarize the team’s overall response to the survey questions. 



Which of the questions is the district prepared to answer? 



Which questions need more investigation and clearer answers? Where might we look 
for this information? 
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Element D: Readiness 



Summarize the team’s overall response to the survey questions. 



Which of the questions is the district prepared to answer? 



Which questions need more investigation and clearer answers? Where might we look 
for this information? 



Element E: Information Quality 



Summarize the team’s overall response to the survey questions. 



Which of the questions is the district prepared to answer? 



Which questions need more investigation and clearer answers? Where might we look 
for this information? 



Element F : System Quality 



Summarize the team’s overall response to the survey questions. 



Which of the questions is the district prepared to answer? 



Which questions need more investigation and clearer answers? Where might we look 
for this information? 



32 — Effective Use of Electronic Data Systems 



Learning Point Associates 









Discussion/Facilitation Worksheet 2 — Planning Next Steps 



Directions: As a group, fill out this planning document for use in moving forward with exploring 
readiness to use an electronic data system. First, indicate the area(s) needing additional exploration. 

Then use the rest of the planning grid to lay out the details of the next steps. Finally, answer the questions 
at the bottom of the page. 



Specific Next Steps 



Areas of Additional Exploration 


Timeline 


Person(s) 

Responsible 


Source(s) of Information 











































How will we sustain this effort? How will we share this infonnation and move forward? 
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Data Use Resources 



Discussion 

Arzberger, P., Schroeder, P., Beaulieu A., Bowker, G., Casey, K., & Laaksonen, L., et al. (2004). 
An international framework to promote access to data. Science, 303(5665), 1777-1778. 
Retrieved July 31, 2006, from http://www.sciencemag.org/cgi/content/summary/ 
303/5665/1777 

Summarizes the current situation with respect to data access, data sharing, and data 
for decision support in the international scientific community. 

Bailey, J. (1996). After thought: The computer challenge to human intelligence. New York: 

Basic Books. 

The core capacities of quantitative data are to tell us about place, pace, and pattern: 
where we are, whether we are moving and how fast, whether we are leaving anyone 
behind. As data tools and technologies improve, we to listen better. This requires new 
habits of mind. 

Data Quality Campaign. (2005). Creating a longitudinal data system: Using data to improve 
student achievement. Austin, TX: National Center for Educational Accountability. 
Retrieved July 31, 2006, from http://www.dataqualitvcampaign.org/files/Publications- 
Creating Longitudinal Data System.pdf 

Describes and briefly analyzes each of the 10 essential elements of a state 
longitudinal data system, most of which also apply to district systems. Includes a 
brief overview of data system fundamentals. 

Huff, D. (1954). How to lie with statistics. New York: Norton. 

Classic review of how to use (and misuse) numerical data. 

Platt, J. (1964). Strong inference. Science, 146, 347-353. 

Early conceptual statement of the importance and power of data and incremental 
experimentation by practitioners. Platt argues that strong inferences about next steps 
are best powered by frequent, small-scale, trial-and-error experiments that rule out at 
least one possible “cause.” He reminds us this requires real rigor in thought and 
action. 

Powers, R. (1999, April 18). Eyes wide open. New York Times Magazine, pp. 80-83. 

Argues that the rise of the experimental method is the core “conceptual revolution 
that lies at the heart of what has happened to the world in these last 1,000 years.” 

Tukey, J. W. (1962). The future of data analysis. Annals of Mathematical Statistics, 33(1). 

Advice for a discipline at the prime of a career from a master in teasing meaning from 
numerical data. 
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Tukey, J. W. (1986). Sunset salvo. American Statistician, 40(1), 72-76. 

Advice at the end of a career from a master: reduce dependence on assumptions, be 
prepared to accept that assumptions are wrong, increase impact of results for decision 
makers, take functional models (those that specify clear relationships) more seriously 
than stochastic ones (those that accept “random” relations among the data). Asking 
good questions that data can meaningfully address will become more difficult, not 
less. 

Surveys 

Consortium for School Networking, (n.d.). How ready are you? A special tool for school district 
leaders to assess readiness to use data effectively. Retrieved July 31, 2006, from 
http://www.3d2know.org/assessment/survey.cfm 

Brief 10-item Web survey about readiness for data-driven decision making. Also 
available are a set of readings, links, and other resources. 

Process Guides, Checklists 

Learning Point Associates. (2006). Information systems for school improvement: Resources. 
Retrieved July 31, 2006, from http://www.learningpt.org/page.php?pageID=31 

A website focused on school-level data analysis tools and resources to support better 
data use. The Data Exploration video is an excellent starting point. 

National Forum on Education Statistics. (2004). Forum guide to building a culture of quality 

data: A school and district resource (NFES 2005-801). Washington DC: National Center 
for Education Statistics. 

Brief document prepared by state-level data and data systems specialists, supporting 
the building of a continuous local culture of quality data and data use. 
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Data Systems Resources 



Discussion 

College of Education, University of Texas at Austin, (n.d.) Data use: Improving education 

practice through data use. Retrieved July 31, 2006, from http://edadmin.edb.utexas.edu/ 
datause/index.htm 

Maintained by Professor Jeff Wayman, this website is a rich resource on data use and 
commercial data products, including screenshots and comparisons. 

Jackl, A., & King, S. (2006). What’s really “in store” for your data warehouse? Austin, TX: 
ESP Solutions Group. 

Too much emphasis can be placed on the data warehouse at the expense of the other 
components of a district’s information system. A well-designed and implemented 
data warehouse can be quite powerful. But just as with any tool, it needs to be used 
for the right purpose. If an agency administrator is sold a data warehouse just to 
satisfy simple reporting needs, then resources have probably been wasted. If a data 
warehouse is overburdened with archive and reporting uses, then it will be unable to 
truly function as needed. 

King, S., & Jackl, A. (2006). What’s behind your data warehouse? Austin, TX: ESP 
Solutions Group. 

Excellent overview of issues in designing a data warehouse for education, including 
scope: figuring out information needs and designing to meet them; reality: being 
practical about the questions to be answered and the data to be consolidated; and 
scalability: ensuring that the warehouse can grow to meet increased demand 
throughout the district but also additional data sets for better analysis and answers to 
questions that were not originally planned for. 

Ligon, G. D. (2005). Data management strategy for states and districts. Austin, TX: ESP 
Solutions Group. 

The yield from data and data systems should be high. Ligon’ s mantra — get the right data, 
get those data right, get them right away, get them the right way, manage the data right — 
clarifies numerous issues. 

Sanniento, J. (2003). Technology tools for the analysis of achievement data: An introductory 
guide for educational leaders. Philadelphia: Laboratory for Student Success, Temple 
University. Retrieved July 31, 2006, from http://www.nwrel.org/scpd/sslc/ 
federal grantees/cohort2/data institutes/binder/resources/C2DataTechToolsforAnalysis.pdf 

Brief introduction to data systems for education, data analysis, and vendor products 
(in 2003). 
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Stein, M. (2003). Making sense of the data: Overview of the K-12 data management and 
analysis market. Retrieved July 31, 2006, from http://www.eduventures.com/ 
about/press room/ll 18 Q3.cfm (required log in) 

Thorough review of the marketplace for student information systems (as of 2003), the 
players, and their distribution. 

Wayman, J. C., Stringfield, S., & Yakimowski, M. (2004). Software enabling school 

improvement through analysis of student data. Baltimore: Center for Research on the 
Education of Students Placed At Risk (CRESPAR), Johns Hopkins University. Retrieved 
July 3 1 , 2006, from http://www.csos.ihu.edu/crespar/techReports/Report67.pdf 

Excellent introduction to the commercial and not-for-profit products available, 
including comparisons and evaluations of a select number of products. 

Surveys 

American Productivity and Quality Center, (n.d.). Information technology within the 
organization [Online survey]. Retrieved July 31, 2006, from 
http://www.apqc.org/portal/apqc/site7pathWresearch/bmrn/osbc/it/index.html 

Information technology organizations worldwide face the same challenge: doing 
more with less. APQC offers in-depth IT benchmarking and best practices that 
compare organizational processes and practices to relevant peers and world-class 
organizations in terms of costs, productivity, efficiency, and cycle time. Not 
education specific. 

Andrews University. (1999). Data warehousing readiness assessment. Retrieved July 31, 2006, 
from http://www.andrews.edu/ITS/AS/dw/Andrews/Ouestions.html 

A 60-item readiness survey, in the form of a simple list of questions. 

Process Guides, Checklists 

Ariyachandra, T., & Watson, H. (2006, January). Benchmarks for BI [business intelligence] and 
data warehousing success. DM Review. Retrieved July 31, 2006, from 
http://www.dmreview.com/article sub.cfm?articleID= 1044330 

Describes metrics used to assess the success of data warehouse architectures and 
reports benchmark data for the metrics and their use. Focused on architectures applied 
outside education; however, the metrics and benchmarks may be readily adapted. 

Assessment & Accountability Comprehensive Center. (2006). Data tool selection criteria: 
Questions to consider when evaluating data tools. Retrieved July 31, 2006, from 
http://datause.cse.ucla.edu/datatool.html 

The Assessment & Accountability Comprehensive Center (whose Web address is 
http://www.aacompcenter.org) hosts a growing number of tools and resources for 
accountability. Under development is a series of supports for data-driven decision 
making and accompanying tools. 
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Mclntire, T. (2004, May 15). Student information systems demystified. School CIO. 

Retrieved July 31, 2006, from 

http://www.schoolcio.corn/shared/printableArticle.ihtml?articleID= 19400338 

Six points that simplify comparing commercial data systems, and brief comparisons 
of 10 systems. 

Mclntire, T. (2004, August 16). Eight buying tips: Data warehouses. Technology & Learning, 
25(1). Retrieved July 31, 2006, from 

lntp://www. tech learn in».com/storv/show Article. ihtml‘?articlclD=26806926 

Offers eight questions and seven attributes as guides through the process of selecting 
a product or finalizing a product design. 

National Forum on Education Statistics. (2005). Forum guide to education indicators (NFES 
2005-802). Washington DC: National Center for Education Statistics. 

Prepared by state-level data and data systems specialists, this document provides 
detailed definitions, calculation formulas, and interpretations for the most common 
quantitative indicators used by schools, districts, and states. 

Schools Interoperability Framework Association. (2003). Implementation toolkit. Retrieved July 
3 1 , 2006, from http://www.sifinfo.org/tool kit.asp 

Describes, in detail, how to plan for the implementation of Schools Interoperability 
Framework requirements. 
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